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Introduction 


The  purpose  of  this  research  has  been  to  develop  a  mobile  motion  capture  system  that  is  capable  of 
quantifying  a  person’s  sense  of  balance,  which  can  in  turn  be  used  as  a  proxy  for  assessing  concussion  or 
musculoskeletal  injury.  During  this  research  period,  we  have  developed  (1)  a  baseline  system  based  on 
the  Microsoft  Kinect  motion  tracker,  (2)  a  graphical  user  interface,  and  (3)  algorithms  of  automated 
balance  scoring.  Furthermore,  we  have  begun  the  process  of  testing  with  human  volunteers  and  used  our 
preliminary  data  to  quantify  system  calibration  and  limitations  of  performance.  We  have  also  compared 
our  system’s  performance  against  that  of  human  experts  and  determined  that  our  system  is  in  general 
more  reliable.  This  report  describes  performance  during  the  first  of  three  years. 

Keywords 

motion  tracking,  balance  assessment,  Microsoft  Kinect,  concussion  assessment 

Accomplishments 

What  were  the  major  goals  and  objectives  of  the  project? 

The  purpose  of  this  project  is  to  create  a  portable  system  for  assessing  balance  in  armed  forces  personnel 
that  can  be  administered  in  the  field  with  minimal  training.  Although  there  are  many  reasons  for  assessing 
an  individual’s  sense  of  balance,  our  project  focuses  on  balance  deficits  caused  by  concussion,  traumatic 
brain  injury,  and  musculoskeletal  injury,  since  these  are  especially  relevant  to  fitness  for  duty.  Our 
deliverable  will  be  a  stand-alone  system  comprising  a  Microsoft  Kinect  motion  tracking  system  and  a 
dedicated  laptop  personal  computer  running  custom  software  for  data  acquisition  and  analysis.  The 
system  is  called  the  Automated  Assessment  of  Postural  Stability,  or  AAPS. 

The  project  is  designed  around  four  Specific  Aims,  or  goals: 

1.  Develop  Baseline  AAPS  System 

2.  AAPS  Calibration  and  Baseline  Evaluation 

3.  AAPS  Field  Evaluation 

4.  Develop  Expanded  xAAPS  Test 

What  was  accomplished  under  these  goals? 

Year  1  was  devoted  to  building  the  baseline  AAPS  system,  calibrating  and  testing  it,  and  establishing  and 
beginning  balance  testing  with  human  subjects.  The  AAPS  system  is  essentially  a  completely 
computerized  version  of  the  Balance  Error  Scoring  System  (BESS)  which  is  a  standardized  test  for 
assessing  balance  that  is  administered  and  scored  by  a  human  expert.  The  software  for  the  baseline  system 
is  complete,  including  the  design  of  a  graphical  user  interface,  and  a  back-end  system  for  storing  and 
organizing  test  data.  In  addition  to  developing  the  baseline  system,  we  have  also  designed  the  specific 
data  collection  protocol,  and  received  approval  from  both  Temple  University  IRB  and  ORP-HRPO.  We 
have  initiated  data  collection  with  healthy  subjects  in  two  phases.  In  the  first  phase,  subjects  performed  a 
series  of  generic  kinematic  movements  and  balance  assessments  while  being  tracked  simultaneously  by 
the  Kinect  and  a  state-of-the-art  motion  capture  system.  This  allowed  us  to  quantify  the  limitations  of  the 
Kinect’ s  motion  tracking  ability,  which  will  be  critical  for  calibrating  the  AAPS  in  the  future.  The  second 
phase  of  subjects  are  our  “core”  group  who  perform  the  BESS  balance  test  while  the  AAPS  system 
records  and  scores  performance.  These  “core”  data  are  also  being  used  to  compare  human  BESS  scoring 


to  computerized  AAPS  scoring  in  order  to  determine  AAPS  performance  relative  to  human  inter-rater 
agreement. 

We  were  also  successful  in  establishing  a  working  relationship  with  the  Temple  University  ROTC  and 
with  caregivers  at  the  Philadelphia  VA  Medical  Center.  We  conducted  three  in-person  meetings  (2 
February,  2  May,  and  23  May  2016)  to  explain  the  project,  solicit  input,  and  establish  the  parameters  for 
working  together  in  the  future  for  data  collection  and  field  testing. 

Technical  Details 


The  AAPS  system  has  been  developed  in  the  C#  programming  language  in  the  Microsoft  Visual  Studio 
2015  programming  environment  and  the  .NET  framework.  The  system  includes  a  comprehensive 
graphical  interface  (GUI)  to  guide  the  operator  and  the  subjects  through  the  BESS  test.  It  also  provides 
user  controls,  data  management  features,  a  real-time  display  of  the  detected  body  and  intuitive  visual 
feedback  on  the  AAPS  tracking  capabilities.  Furthermore,  the  GUI  has  been  designed  to  be  user-friendly 
and  its  use  only  requires  minimal  training  and  experience.  The  current  AAPS  GUI  is  shown  in  Figure  1. 

The  AAPS  system  utilizes  an  inexpensive  Microsoft  Kinect  v2.0  motion  capture  sensor  and  a  custom 
designed  software  suite  for  Microsoft  Windows  to  objectively  track  kinematics  during  the  BESS  test.  Set¬ 
up  time  is  minimal,  and  no  additional  calibration  time  is  required.  These  features  make  the  AAPS  a 
valuable  balance  assessment  tool  to  be  utilized  in  the  field.  It  only  requires  the  patient  to  assume  the  right 
stance  and  the  system  starts  extracting  tracking  information  from  the  sensor  data  stream  and  uses  a  GUI  to 
display  the  tracked  skeleton.  By  using  some  simple  visual  feedback  features,  the  system  provides  the  user 
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Figure  1 :  The  AAPS  Graphical  User  Interface  showing  both  run-time  (top)  and  results  (bottom)  screens. 


with  information  regarding  the  subject’s  position,  joints  and  eyes  detection.  This  helps  the  non-trained 
user  in  positioning  the  subject  for  the  test  and  guarantees  that  all  the  necessary  parameters  are  tracked 
correctly  before  starting  the  test.  During  the  test,  in  each  frame,  the  AAPS  extracts  two  types  of  data  from 
the  Kinect  sensor:  infrared  camera  videos,  which  will  be  saved  in  the  computer  memory  for  optional  off¬ 
line  analysis;  and  the  subject’s  joint  location  and  eye  data.  These  will  be  used,  frame-by-frame,  to  build 
arrays  containing  the  three  metrics  that  will  be  then  used  to  evaluate  errors,  namely  relative  joint 
distances,  segment  angles  and  eye  state.  After  test  completion,  these  metrics  will  be  used  to  compute 
balance  errors. 

The  performance  of  the  Kinect™  2.0  as  a  tool  to  evaluate  kinematic  variables,  as  compared  to  current 
standard  methods,  is  a  subject  of  great  interest.  Although  several  studies  have  been  published  in  this  area, 
the  general  trend  has  been  to  compare  motion  capture  (MOCAP)  systems  based  on  scalar  summary 
measures  of  some  ad-hoc  selected  metric.  Examples  from  previous  work  include  excursion  range,  mean 
or  peak  displacement,  mean  velocity,  and  timing  of  discrete  signal  events.  While  such  metrics  are 
commonly  studied  in  biomechanics,  they  do  not  adequately  quantify  the  temporal  structure  of  the  signals 
under  comparison,  and  are  thus  limited  in  terms  of  the  generalizability  of  their  results.  Considering  these 
limitations,  it  is  clear  that  a  more  thorough  evaluation  of  Kinect  2.0  raw  data  performance  as  a  MOCAP 
tool  and  its  validation  against  a  gold-standard  3D  system  are  needed.  Specifically,  we  have  validated  the 
Kinect  against  a  professional  Qualisys  three-dimensional  motion  capture  system  with  12  IR-cameras.  In 
order  to  better  analyze  the  Kinect-based  MOCAP  performance  in  evaluating  human  kinematics,  we  chose 
a  complete  range  of  dynamic  movements  and  clinical  tests  that  can  be  used  as  broad  indicators  of 
functional  movements.  We  acquired  data  from  four  healthy  subjects  in  a  professional  motion  analysis 
laboratory  in  which  signals  from  two  side-by-side  Kinects  were  recorded  simultaneously  with  data  from  a 
professional  grade  Qualisys  system.  The  side-by-side  Kinect  recordings  were  made  to  quantify  sensor 
reliability. 

Four  subjects  (three  males,  mean  age  23)  performed  a  series  of  movements  while  being  simultaneously 
tracked  by  two  immediately  adjacent  Kinect  2.0s  and  a  professional-grade  motion  capture  system 
(Qualisys,  Gothenburg,  Sweden).  The  two  Kinects  were  used  to  evaluate  inter-unit  accuracy,  and  motion 
tracking  results  from  both  systems  were  compared  to  the  gold  standard  Qualisys  system.  The  movements 
and  the  recording  paradigm  were  specifically  designed  to  facilitate  the  quantification  of  errors  in  tracking 
joint  angles  and  limb  locations  relative  to  all  three  cardinal  planes  of  the  body  axis:  coronal,  sagittal,  and 
transverse.  The  four  subjects  were  asked  to  move  through  a  series  of  postures  twice.  Each  posture  was 
preceded  with  a  large  movement  such  as  a  T-pose  or  an  overhead  reach  in  order  to  facilitate  time- 
synchronization  of  the  Qualisys  and  Kinect  systems;  these  movements  were  not  scored  or  otherwise 
included  in  the  results.  Three  subjects  performed  a  battery  of  standard  clinical  tests  of  dynamic  posture, 
whereas  the  fourth  subject  performed  the  stereotyped  postures  (e.g.  movements  restricted  to  a  single 
plane). 

Figure  2  shows  representative  joint  center  displacements  of  head,  spine,  left  hip,  and  right  hip,  as 
measured  for  a  single  subject  during  a  sit-to-stand  test;  data  from  both  Qualisys  and  Kinect  are  displayed. 
This  figure  illustrates  similarities  between  the  joint  displacement  data  are  from  the  two  motion  capture 
systems.  This  figure  also  underscores  the  need  for  multiple  similarity  metrics,  since  it  is  possible  to  have  a 
high  cross  correlation  but  also  a  high  mean  squared  error.  The  poorest  cross-correlation  scores  were 
generally  obtained  with  respect  to  those  axes  along  which  the  observed  motion  was  negligible 
(mediolateral  in  the  case  of  sit-to-stand).  Note  also  that,  in  general,  tracking  between  the  two  systems  was 
less  robust  in  the  anteroposterior  plane  (representing  ‘depth’  away  from  the  Kinect  sensor),  than  in  the 
rostrocaudal  plane  (vertical)  plane. 


We  used  two  separate  error  metrics  to  describe  the  performance  of  the  Kinect  system  relative  to  the 
benchmark  Qualisys  system.  First,  we  calculated  cross-correlation  coefficients,  which  effectively  describe 


ML  [cm]  RC  [cm]  AP  [cm]  ML  [cm]  RC  [cm]  AP  [cm] 


how  much  information  one  signal  can  yield  about  another,  but  that  is  generally  blind  to  errors  of  constant 
or  near-constant  offset  or  bias.  Secondly,  we  calculated  the  mean-squared  errors  (MSEs)  between  the 
various  output  signals.  MSE  measures  constant  or  near  constant  differences  between  signals  but  is  blind 
to  signal  correlation. 
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Figure  2:  Head,  spine  middle,  left  and  right  hip  joint  center  displacements  in  cm  as  derived  from  both  MOCAP  systems  Qualisys 
(yellow)  and  Kinect  (blue),  during  a  sit  to  stand  test.  The  displayed  signals  are  for  a  single  subject  and  a  single  trial.  Values  in 
the  lower  left  corner  of  each  plot  show  the  cross-correlation  coefficients  and  the  mean  squared  error  calculated  between  the 
Qualisys  and  the  Kinect  displacements  over  time.  ML  =  Mediolateral,  RC  =  rostrocaudal,  AP  =  anteroposterior 


While  Figure  2  shows  tracking  in  a  specific  case,  Table  1  summarizes  the  cross-correlation  and  mean 
square  error  of  the  3d  movement  tracking  across  12  different  movements.  Note  that  the  error  depends  on 
the  movement  and  the  plane.  These  results  accurately  capture,  for  the  first  time,  the  tracking  resolution  of 
the  Kinect  with  respect  to  movements  and  postures  that  are  relevant  to  the  physical  therapy  community. 
Importantly,  our  observation  that  the  errors  vary  according  to  the  specific  movement  have  ramifications 
for  how  the  AAPS  is  calibrated.  Fortunately,  the  data  (which  here  are  only  summarized;  a  full  treatment 
will  be  presented  in  our  forthcoming  journal  publication)  demonstrate  sufficient  accuracy  to  justify  using 
the  Kinect  as  the  motion  tracking  for  the  AAPS  system. 

Progress  Relative  to  Goals 

At  the  end  of  Year  1  (of  3),  this  project  is  ahead  of  schedule  and  under  budget.  Expenditures  during  Year 
1  totaled  $384,000,  versus  a  budgeted  amount  of  $442,496.  The  underspending  was  a  results  of  two 
factors.  First,  it  took  several  months  to  staff  up  completely.  Postdoc  Napoli  started  on  1  November  2015, 
and  Postdoc  Glass  on  1  January  2016.  Although  graduate  research  assistant  Ward  began  work  at  the  start 
of  the  project,  he  only  devoted  50%  effort  until  1  January  2016.  Secondly,  graduate  RA  Ward  required 
less  tuition  than  was  originally  budgeted,  because  he  has  completed  didactic  coursework  and  is  now 
devoted  primarily  to  research. 

Despite  being  under  budget,  the  project  is  ahead  of  schedule.  Table  2  shows  the  status  of  the  Specific 
Aims  and  Sub- Aims,  relative  to  the  timeline  as  originally  stated  in  the  research  proposal.  As  stated  earlier, 
the  baseline  AAPS  has  been  completely  built  and  tested,  and  work  human  subject  evaluation,  which 
wasn’t  slated  to  begin  until  Year  2,  is  already  well  under  way.  We  have  even  made  initial  progress  into 
feasibility  and  planning  for  the  xAAPS  development,  which  is  the  final  Aim  and  the  subject  of  Year  3. 
Note  that,  although  Aim  1.3  is  not  entirely  complete,  we  expect  it  to  be  closed  out  by  the  end  of  next 
quarter. 


Table  1 :  Errors  in  tracking  joint  displacement.  Errors  are  averaged  over  all  repetitions  of  all  20  tracked  body  points.  AP  = 

anteroposterior,  RC  =  rostrocaudal,  ML  =  mediolateral 


AP 

error 

RC 

ML 

4.30 

4.51 

2.63 

10.13 

4.41 

5.07 

9.50 

5.40 

2.18 

4.27 

5.72 

1.86 

2.17 

2.05 

1.86 

6.15 

3.54 

3.84 

1.24 

1.98 

0.98 

1.95 

3.02 

1.10 

1.85 

2.61 

2.85 

1.58 

2.24 

3.88 

1.46 

4.09 

1.76 

2.16 

3.21 

3.10 

Movement 

correlation 

AP  RC  ML 

sit  to  stand 

0.85 

0.84 

0.67 

timed  up  and  go 

0.97 

0.75 

0.83 

alternating  barbell  lunges 

0.92 

0.80 

0.87 

overhead  squats 

0.78 

0.86 

0.61 

marching  in  place 

0.88 

0.80 

0.92 

time  to  stabilization 

0.93 

0.89 

0.87 

shoulder  ab/adduction 

0.60 

0.75 

0.65 

shoulder  flexion/extension 

0.66 

0.65 

0.64 

hip  ab/adduction 

0.71 

0.78 

0.95 

combined  hip/knee  flexion/extension 

0.85 

0.87 

0.97 

combined  arm  ab/adduction 

0.49 

0.55 

0.56 

sagittal/frontal  trunk  leans 

0.78 

0.59 

0.84 

Table  2:  Project  status  relative  to  timeline  originally  stated  in  the  research  proposal. 


Specific  Aim  1  -  Develop  AAPS  Baseline  System 

Port  Image  Processing  Code  to  C/C++ 

1-5  months 

100% 

Develop  User  Interface 

4-8  months 

100% 

Develop  AAPS  for  Field  Use 

7-12  months 

75% 

Specific  Aim  2  -  AAPS  Calibration  and  Baseline  Evaluation 

Healthy  Subject  Evaluation 

12-18  months 

35%  (17  subjects  complete) 

Concussion  Subject  Evaluation 

18-30  months 

0% 

Mild  Musculoskeletal  Injury  Subject 
Evaluation 

18-30  moths 

0% 

Specific  Aim  3  -  AAPS  Field  Evaluation 

Evaluate  use  by  non-clinician  operators 

12-15  months 

0% 

Evaluate  AAPS  in  Field  Conditions 

14-24  months 

0% 

Specific  Aim  4  -  Develop  Expanded  xAAPS  Test 

Determine  movements  for  xAAPS  test 

18-22 

15% 

Update  AAPS  software  for  xAAPS  test 

18-30 

10% 

Evaluate  xAAPS  test 

30-36 

0% 

What  opportunities  for  training  and  professional  development  did  the  project  provide? 

This  project  has  provided  excellent  opportunities  for  training  and  professional  development,  since  almost 
all  of  the  main  work  has  been  performed  by  trainees.  Although  the  co-PIs  retain  close  oversight,  day  to 
day  operations  and  planning  have  been  delegated  to  the  postdoctoral  fellows,  for  whom  this  is  excellent 
professional  training.  The  graduate  research  assistant  has  been  mentored  by  both  PI  Obeid  and  the 
Fellows.  His  responsibilities  have  included  writing  software  and  developing  much  of  the  back  end 
mathematics  behind  the  manipulations  of  the  three  dimensional  mathematics.  The  two  undergraduates 
(both  female;  one  underrepresented  minority)  have  learned  to  program,  to  debug  software,  and  to  work 
use  team-based  software  tools  such  as  bug  tracking  and  distributed  version  control. 


Both  postdoctoral  fellows  and  the  graduate  student  have  attended  research  conferences  this  year  and  they 
have  all  contributed  to  planning  and  writing  three  manuscripts,  the  first  of  which  will  be  submit  for 
review  within  a  month. 


How  were  the  results  disseminated  to  communities  of  interest? 


Most  of  Y ear  1  has  been  devoted  more  to  technology  development  and  less  to  dissemination.  Even  so,  the 
team  has  made  excellent  progress  in  disseminating  our  work.  The  team  published  a  paper  at  the  IEEE- 
EMBS  2016  Annual  Conference,  which  was  presented  by  Dr.  Napoli.  The  team  is  also  preparing  three 
manuscripts,  the  first  of  which  will  be  submit  for  review  within  one  month.  These  three  papers  address: 

1.  Techniques  for  calibrating  the  Kinect  and  using  it  for  motion  capture  in  a  physical  therapy 
paradigm 

2.  BESS  test  scored  using  AAPS/Kinect  versus  motion  tracking  from  professional-grade  system 

3.  BESS  test  scoring  using  AAPS  versus  human  scoring 

The  second  two  manuscripts  are  expected  to  be  submit  during  the  second  quarter  of  Year  2. 

What  do  you  plan  to  do  during  the  next  reporting  period  to  accomplish  the  goals  and 
objectives? 

During  Year  2,  we  expect  the  emphasis  to  be  on  data  collection  and  analysis.  This  data  collection  will 
occur  both  in  the  lab  and  in  the  field,  in  collaboration  with  Temple  ROTC  and  our  military  partners.  The 
team  will  also  harden  the  technology  for  field  use  by  minimally  trained  non-medical  cadets.  During  the 
latter  half  of  the  year,  we  will  begin  the  process  of  developing  the  expanded  xAAPS  system,  both  in  terms 
of  defining  movements  to  be  tracked  and  writing  software  to  track  and  evaluate  those  movements. 

Impact 

Principal  project  discipline 

Although  there  have  been  a  handful  of  reports  in  the  literature  describing  how  accurate  the  Kinect  is,  none 
of  them  have  adequately  taken  into  account  either  the  range  of  ‘normal’  movements  (in  a  kinematic  sense) 
or  the  allocation  of  errors  into  the  three  cardinal  planes.  Such  data  is  critical  in  order  to  understand  the 
limits  of  accuracy  that  can  be  expected  from  Kinect-based  systems.  By  publishing  our  findings  in  this 
area,  we  are  making  a  fundamental  contribution  to  the  Kinect-based  motion  tracking  research  community. 

Other  disciplines 

Nothing  to  report. 

Technology  transfer 
Nothing  to  report  (yet). 

Society  Beyond  Science  and  Technology 

The  goal  of  this  project  has  always  been  to  improve  awareness  and  treatment  of  concussion  by  providing 
a  low  cost,  low  complexity  way  of  quantifying  degradation  in  balance  ability.  Given  what  is  now  known 
about  the  pervasiveness  of  concussion  and  mild  brain  injury,  especially  in  the  armed  forces  community, 
this  project  has  the  potential  to  contribute  positively  to  societal  health  and  wellbeing. 


Changes/Problems 

Changes  in  approach  and  reasons  for  change. 

Nothing  to  report. 

Actual  or  anticipated  problems  or  delays  and  actions  or  plans  to  resolve  them. 

Nothing  to  report. 

Changes  that  have  a  significant  impact  on  expenditures. 

Nothing  to  report. 

Significant  changes  in  use  or  care  of  human  subjects,  vertebrate  animals,  biohazards, 
and/or  select  agents. 

Nothing  to  report. 

Products 

[1]  Napoli  A,  Glass  S,  Ward  C,  Tucker  C,  Obeid  I  (2017)  “Implementation  and  validation  of  a 
generalized  motion  capture  system  using  Microsoft  Kinect  2.0,”  in  preparation  for  submission  to 
Biomedical  Signal  Processing  &  Control 

[2]  Napoli  A,  Ward  C,  Glass  S,  Tucker  C,  Obeid  I  (2016)  “Automated  Assessment  of  Postural  Stability 
System,”  IEEE  Engineering  in  Medicine  &  Biology  Conference,  Orlando,  FL. 


Participants  &  Other  Collaborating  Organizations 

What  individuals  have  worked  on  the  project? 


Name: 

Project  Role: 

Person-Months: 

Contribution: 


Name: 

Project  Role: 

Person-Months: 

Contribution: 


Iyad  Obeid,  PhD 
co-Principal  Investigator 
3 

Dr.  Obeid  contributed  to  project  design  and  management,  analyzed  data, 
supervised  data  marshalling,  wrote  quarterly  reports,  and  contributed  to  all 
technical  publications. 

Carole  Tucker,  PhD 
co-Principal  Investigator 
3 

Dr.  Tucker  contributed  to  project  design  and  management,  IRB  preparation, 
human  subject  protocol  design,  data  collection,  and  analysis,  and  all  technical 
publications. 


Name: 

Project  Role: 

Person-Months: 

Contribution: 


Name: 

Project  Role: 

Person-Months: 

Contribution: 


Name: 

Project  Role: 

Person-Months: 

Contribution: 


Name: 

Project  Role: 

Person-Months: 

Contribution: 

Name: 

Project  Role: 

Person-Months: 

Contribution: 


Alessandro  Napoli 
Postdoctoral  Fellow 
10 

Was  responsible  for  managing  all  aspects  of  the  software  organization  and 
development,  and  contributed  heavily  to  actual  software  creation.  He  managed 
the  graduate  RA  and  the  undergraduates,  contributed  to  data  collection  and 
analysis,  and  took  a  leading  role  on  all  technical  publications. 

Stephen  Glass 
Postdoctoral  Fellow 
8 

Was  responsible  for  managing  all  aspects  of  data  planning,  collection  and 
analysis,  including  IRB  development.  He  managed  junior  students,  and  also 
developed  our  analysis  of  human-versus-Kinect  balance  scoring  analysis.  Dr. 
Glass  also  took  a  leading  role  in  all  technical  publications. 

Christian  Ward 

Graduate  Research  Assistant 

5 

Was  responsible  for  software  development  and  testing,  and  developed  all  3D 
vector  mathematics.  Also  worked  closely  with  the  undergraduate  researchers. 

Orlena  Roe 

Undergraduate  Researcher 
3 

Contributed  to  software  testing  and  development  of  the  graphical  interface. 

Elizaveta  Ibeme 
Undergraduate  Researcher 
1 

Contributed  to  software  testing  and  wrote  software  tools  for  data  analysis. 


Has  there  been  a  change  in  the  other  active  support  of  the  PD/PI(s)  or  senior/key  personnel 
since  the  last  reporting  period? 

Nothing  to  report. 

What  other  organizations  have  been  involved  as  partners? 

Nothing  to  report. 

Special  Reporting  Requirements 

See  Quad  Chart  in  the  Appendix 

Appendices 

Quad  Chart  -  see  next  page 


Automated  Assessment  of  Postural  Stability  (AAPS) 

Log  Number:  MR141272 
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